The demand for ad-hoc and real-time data analyses by many users in parallel is increasing in the same way that the data quantities to be processed are increasing. To keep pace with the competition in the current economic climate, it is crucial to have the latest information about processes within an organization (e.g. a company) and/or in the market to be able to make the right decisions promptly. Therefore, it is important that the queries generated by analysis tools are processed as quickly as possible. The limiting factor in fulfilling these requirements is typically access to external memory because of the large gap between the access times in the main memory and to external storage. Various approaches to solving this problem can include restricting the data to be read from external storage to a minimum and fastest possible access, avoiding input and output accesses by holding and processing data in the main memory, and parallel processing of queries distributed across a number of suitable server systems (also referred to as “blades”).
Management of all data in the main memory can reach an upper limit on practicality relatively quickly, particularly in data-intensive applications, for example including but not limited to business software solutions such as enterprise resource planning (ERP) systems. System memory (e.g. random access memory or RAM) is generally more expensive compared to disk space and can also be limited to an upper bound by the expansion capabilities of a system. As such, suitable compression techniques that both reduce the data volume and allow efficient use of individual tuples and fields are very useful. Reductions in the size of data units that are processed can also provide additional speed benefits via improved utilization of CPU caches and reduction in the amount of data requiring processing.
Online transaction processing (OLTP) and online analytical processing (OLAP) are two potential applications in which high speed data processing may be useful. OLTP is typically used in a company's operational day-to-day business, in which many small and short transactions are executed in parallel and high throughput is required. Databases optimized for write access, such as a typical relational database management system (DBMS) with data organization in rows (row store), can be suitable for very detailed transaction data that must be up-to-date. In contrast, the analysis of operational data over a longer period of time to support decision-making is the more typical focus of OLAP operations. In general, OLAP queries are more complex and the level of detail required for the results is lower. The longer-running transactions associated with OLAP generally access only some of the columns in read mode. As such, a database optimized for read access can be beneficial for support of OLAP processes.